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SECTION  1 
INTRODUCTION 

This  paper  describes  an  algorithm  for  solving  the  general 
< nonlinear  programming  problem.  The  method  is  an  extension  of  the 

accelerated  multiplier  method  described  in  Ref.  1.  The  algorithm 
combines  aspects  of  the  multiplier  method  proposed  by  Powell  (Ref . Z) 
and  independently  by  Hestenes  (Ref.  3),  with  the  gradient  projection 
algorithms  suggested  by  a number  of  authors  including  Fletcher  (Ref.  4) 
and  Murtagh  and  Sargent  (Ref.  5).  The  algorithm  deals  with  inequality 
constraints  directly,  while  retaining  the  favorable  numerical  properties 
of  the  multiplier  methods.  Search  directions  are  computed  using  a 
projection-like  formula  which  avoids  the  ill-conditioning  in  penalty 
function  methods  reported  by  Fletcher  and  McCann  (Ref.  6)  and  ad- 
dressed by  Biggs  (Ref.  7)  and  Murray  (Ref.  8).  The  convergence  of 
the  multiplier  method  is  accelerated  by  using  a gradient  projection 
technique  to  solve  the  constraints. 

The  problem  of  interest  in  this  paper  is  to  determine  the  n-vector 
X that  minimizes  the  scalar  function, 

f 

f(x)  = f(Xj,  . . . , xj  (1) 

called  the  objective  function,  subject  to  the  equality  constraints 

c^(x)  = 0,  i = 1 k,  (2)  j 

and  the  inequality  constraints  : 

c^(x)  i 0 i = (k+ 1),  . . . , m.  (3)  | 

The  functions  f(x)  and  c-(x)  are  assumed  continuously  differentiable 

I 

to  second  order  in  the  region 

I 

*L  ^ (4)  I 

where  Xj^  and  xy  are  the  specified  lower  and  upper  bounds.  Bounds  | 

determine  a region  of  computability  and,  unlike  constraints,  cannot  I 

be  violated  during  the  iterative  process. 
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Define  the  Lagrangian  function, 

L(x,  k)  = f(x)  + c'^(x)  X (5) 

where  c(x)  is  the  m-vector  of  all  constraints  and  A is  the  m-vector 
of  Lagrange  multipliers.  At  the  optimum  point  (x*,  A*) 

7L(x*,  A*)  = g(x*)  + G(x*)  A*  = 0 (6) 

where  VL  is  the  gradient  vector  of  the  Lagrangian  fvinction  with 
respect  to  x,  g(x)  is  the  gradient  vector  of  f(x),  and  the  n x m Jacobian 
matrix  is  given  by 

[i>Ci  ...  6c_,*l 


^c  1 , . 


]=  . 


Furthermore, 


l6cj  6 Cm 


Xi  Ci  (X  ) =0 


i“l,  . . .,  m, 


where 


Ai  s 0 


i = (k+1),  . . . , m. 


In  order  to  distinguish  constraints  that  are  active  at  a solution, 
define  the  set 


I i I Cj  (x  ) = 0,  i c|l,  2, . . . m 


called  the  basic  set  of  constraints.  An  estimate  B of  the  basic  set  of 
constraints  shall  be  referred  to  as  a basis.  Clearly  B contains  all 
equality  constraints,  if  the  gradients  of  the  constraints  in  the  basis 
are  linearly  independent  at  the  solution,  then  (6),  (8),  and  (9)  constitute 
the  Kuhn-Tucker  necessary  conditions  for  the  existence  of  an  optimum. 

Most  nonlinear  programming  algorithms  proceed  by  obtaining 
a sequence  of  points  which  either  satisfy  the  criteria  (8)  which  we  shall 
refer  to  as  the  constraint  condition  or  the  condition  (6)  which  we  shall 
refer  to  as  the  Lagrangian  condition.  Projected  gradient  algorithms, 
for  example,  attempt  to  satisfy  the  constraints  at  each  step  while  moving 
toward  satisfaction  of  the  Lagrangian  condition.  In  contrast,  penalty 
function  and  multiplier  methods  attempt  to  satisfy  the  Lagrangian 
condition  and  then  move  toward  constraint  satisfaction.  The  gradient 
projection-multiplier  method  to  be  described  cycles  between  satisfying 
the  Lagrangian  condition  and  the  constraint  condition. 

Points  satisfying  the  Lagrangian  condition  or  the  constraint 
condition  are  located  using  an  unconstrained  optimization  algorithm. 
When  the  algorithm  is  applied  to  the  minimization  of  an  augmented 
objective  function,  a point  satisfying  the  Lagrangian  condition  can  be 
located.  A point  satisfying  the  constraints  can  be  found  by  applying  the 
same  algorithm  to  the  penalty  function  alone.  The  algorithm  requires 
both  function  and  gradient  information  and  is  designed  for  the  class  of 
problems  in  which  the  function  and  gradient  evaluations  are  relatively 
expensive  from  a computational  standpoint. 

Section  2 describes  the  unconstrained  algorithm,  and  Section  3 
discusses  how  the  procedure  is  applied  to  constrained  optimization. 

The  detailed  description  of  the  constrained  optimization  algorithm  is  in 
Section  4.  Section  5 presents  numerical  experience  with  the  approach. 


SECTION  2 

UNCONSTRAINED  OPTIMIZATION  ALGORITHM 

In  this  section,  the  unconstrained  optimization  algorithm 
developed  in  Ref.  1 is  modified  somewhat.  Specifically,  we  are 
concerned  with  the  following  augmented  objective  function: 

J(x,  A.,  r)  = f(x)  + c^(x)  A + rP(x)  = L(x,  A)  + r P(x),  (11) 

where  r is  a scalar  referred  to  as  the  penalty  weight,  A,  is  an  m-vector 
of  estimates  of  the  Lagrange  multipliers,  and  the  penalty  function  is 
defined  by 

P{x)  = E (x)  + r , c.  ^(x),  (12) 

ifB  iC  B 


where 


c^  (x)  = min  [0,  c^  (x)  ], 


and  B the  complement  of  B,  is  the  set  of  constraints  not  in  the  basis. 

Expressions  for  the  first  and  second  derivatives  are  obtained 
by  differentiation  of  (11).  Thus, 


where 


VJ=  VL  + r VP 


VL  = g + GA. 


VP  = E 2c.  (x)  Vc.  (x) 
ieB  ^ 


it 


+ E 2c.  (x)  Vc|  (x) 
ieB' 


The  Hessian  matrix  is 


H = T + r (U+V), 


FRECmNa  FiOK  w.AMy 


(18) 


where  T = V^L, 


U = Z)  Zcj(x)  V^c-(x) 
icB 

and 

V = 2Vci(x)  Vci(x)'^ 
ieB 


i E 2 Ci(x)V  ^Cj  (x), 
ieB' 


+ E ^ 2Vc^  (x)7cj(x)^. 
ieB' 


(19) 


(20) 


When  the  function  J is  approximated  by  a quadratic  function, 
an  estimate  of  the  minimum  point  can  be  obtained  by  locating  the 
minimum  of  the  quadratic  approximation.  The  gradient  at  the  minimum 
point  of  the  approximation  must  necessarily  be  zero,  and  it  can  be 
demonstrated  that  the  gradient  condition  defines  the  system  of  equations 

Hs  = VJ.  (21) 

The  Hessian  matrix  H is  defined  by  (17),  where  the  matrices  T and  U 
are  approximated  using  a rank-one  recursive  formula  originally  stated 
in  Ref.  9 and  specialized  for  least  squares  applications  in  Ref.  10. 

The  matrix  V can  be  evaluated  from  local  gradient  information. 

The  search  direction  vector  s is  usually  obtained  by  solving  the  system 
(21),  and  a new  estimate  of  the  optimum  point  constructed  according  to 

X = x - ps.  (22) 

The  scalar  p is  determined  by  a one-dimensional  search  procedure. 
Thus  the  unconstrained  optimization  algorithm  proceeds  by  taking  a 
series  of  steps  defined  by  (22),  where  at  each  iteration  the  Hessian 
matrix  H is  constructed  as  previously  described,  and  the  direction 
vector  s is  determined  from  (21),  The  current  method  is  unique  in 
that  a different  method  for  the  determination  of  the  search  direction 
vector  is  proposed. 


Define  the  jth  element  of  the  vector  of  constraints  c(x),  as 


Cj  (x)  = C|(x)  if 


where  j = 1.  . . m,  m < m. 


ieB 

or 

ieB  ^ and  c^(x)  0, 


(23) 
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A 

The  m constraints  in  the  vector  c shall  be  referred  to  as  active  con- 
straints. Using  a similar  notation  for  the  corresponding 

A 

gradients,  define  the  Jacobian  matrix  of  active  constraints  G,  and  (14) 
becomes 

A ^ 

VJ  = VL+2rGc  (24) 

where  the  definition  (16)  has  been  used.  In  like  fashion,  if  we  define 

A = T + rU,  (25) 

the  expression  for  the  Hessian  matrix  (17)  is 

H = A + 2r  GG^.  (26) 

Combining  (21),  (24),  and  (26)  one  obtains  the  system  of  equations 


(A  + 2r 


T 

G G ) 


V L + 2r  Gc  , 


Define  the  augmented  system  of  equations 


(27) 


(28) 


where  the  vector  X has  a dimension  equal  to  the  number  of  columns  in 
G or  the  number  of  active  constraints.  I is  the  m x m identify  matrix. 

Let  us  show:  (a)  that  the  vector  s obtained  by  solving  (27)  is 
equal  to  that  obtained  by  solving  (28),  and  (b)  that  the  vector  s obtained 
by  solving  (28)  approaches  the  projected  gradient  direction  as  r becomes 
large.  To  indicate  that  the  solution  of  (27)  is  equal  to  that  obtained  from 
(28),  we  employ  the  formula  for  the  inverse  of  a matrix  in  terms  of 
submatrices: 


-1 
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(29) 


where 


^11  = 

(A  + 2r  G G ) 

^12 

2r  CiiG 

2 'T  ^ 

^22 

4r  G C^^  G-2rl, 

Clearly  from  (28)  and  (29) 

s = (A+2rGG^)‘^  (VL  + 2r  Gc)  (30) 

which  is  the  solution  obtained  from  (27). 

To  investigate  the  limiting  behavior  it  is  convenient  to  expand 
(28)  to  form 


As  + GX  = VL  + 2r  Gc 


(31) 


and 


G 


T 

s 


0. 


(32) 


It  has  been  demonstrated  by  Fiacco  and  McCormick  (Ref.  12). 
that 

A ^ 

lim  2 rc  = X 
r—^oo 


(33) 


where  X is  the  vector  of  Lagrange  multipliers.  Assume  that 
U = 0 (which  is  reasonable  for  large  r),  so  that  A is  independent  of  r. 
Applying  this  limiting  expression,  (31)  becomes 

As  = VL.  (34) 
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Furthermore  we  can  write  (32)  as 


r 


» 

t 

f 


^ T 
G s 


(35) 


Making  use  of  the  definition  of  VL  from  (15)  it  follows  that  (34)  and 
(35)  can  be  written  as 

/a  g\  /g 

\G^  0 


(36) 


Solution  of  this  system  using  the  same  partitioning  formulas  used  in 
(29)  results  in  the  standard  projected  gradient  search  direction. 

Having  considered  the  limiting  behavior,  it  appears  that  there 
is  some  advantage  to  determining  the  search  direction  using  the 
augmented  system  (28)  instead  of  the  system  (27).  Specifically,  the 
condition  number  of  the  augmented  system  should  approach  a constzmt 
value  as  r becomes  large,  since  the  system  (28)  approaches  the 
system  (36).  In  contrast,  Fletcher  and  McCann  (Ref.  6)  report  that 
the  condition  number  of  the  system  (27)  becomes  infinite  as  r increases. 
Thus,  at  the  expense  of  solving  the  larger  system  (28),  the  ill- 
conditioning  associated  with  the  solution  of  (27)  can  be  avoided. 

A second  advantage  of  the  proposed  method  is  the  fact  that  it  is 
unnecessary  to  assume  that  A is  positive  definite.  Consequently,  the 
suggested  approach  is  applicable  when  methods  requiring  inversion 
or  Cholesky  decomposition  of  A are  not.  For  example,  a linear 
objective  function  poses  no  difficulty  in  the  new  method,  since  it  is  not 
necessary  that  A have  full  rank. 

Having  discussed  the  general  procedure  for  computing  the  search 
direction  vector,  let  us  make  some  observations  pertinent  to  specific 
applications.  First,  it  should  be  clear  that  the  general  algorithm  is 
applicable  to  unconstrained  optimization  problems  if  we  set  m = 0 and 
r = 0.  Nonlinear  least  squares  problems  can  be  solved  using  the 
algorithm  if  we  set  f(x)  = 0,  VL  = 0,  and  r = 1 for  m ^ n.  Further- 
more, the  algorithm  can  be  used  to  satisfy  the  constraints  in  con- 

A 

strained  optimization  by  posing  a least  squares  problem  with  m < n 
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and  7L.  = 0.  With  two  exceptions,  the  search  direction  is  determined 
from  (28).  The  first  exception  involves  the  nonlinear  least  squares 
case  when  the  matrix  U = 0.  In  this  instance,  it  is  numerically  preferable 
to  apply  the  linear  least  squares  algorithm  directly  to  (35).  The  second 
exception  occurs  when  solving  constraints  in  a constrained  optimization 
problem  with  U = 0 and  m < n.  In  this  situation  we  solve  (36),  which 
is  the  limiting  form  of  (28). 

It  should  be  noted  at  this  point  th;-  t any  technique  for  solving 
linear  systems  can  be  applied  to  (28).  For  the  class  of  problems  of 
interest  to  the  author,  the  computational  cost  of  evaluating  the  function 
and  gradient  is  far  greater  than  the  cost  of  solving  the  system  of 
equations.  Consequently,  in  the  computer  implementation,  the  system 
(28)  is  solved  using  the  linear  least  squares  procedure  described  in 
Ref.  13. 

To  summarize,  an  algorithm  for  finding  the  unconstrained 
minimum  of  the  augmented  objective  function  (11)  has  been  outlined. 

The  algorithm  consists  of  a sequence,  of  steps  given  by  (22)  in  the 
directions  defined  by  the  vector  s.  The  vector  s is  computed  by  solving 
the  augmented  system  (28)  subject  to  the  exceptions  noted  above.  The 
Hessian  matrices  are  generated  recursively  using  a rank-one  formula 
as  described  in  the  references. 
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APPLICATION  TO  CONSTRAINED  OPTIMIZATION 


Having  developed  a general  unconstrained  optimization  algorithm, 
let  us  consider  its  use  in  an  overall  nonlinear  programming  method.  It 
has  been  established  that  a point  which  minimizes  the  augmented  objective 
fvinction  (11)  also  satisfies  the  Lagrangian  condition  (6)  for  specified 
penalty  weight  r and  multipliers  X.  It  was  indicated  above  that  a point 
satisfying  the  constraint  condition  (8)  could  be  determined  by  minimizing 
J,  provided  that  we  set  Li(x,  \)  = 0 and  r = 1.  Since  for  m < n there  are 
fewer  constraints  than  variables,  in  general,  there  is  no  unique  solution 
to  the  constraints.  In  order  to  make  the  point  on  the  constraint  surface 
unique,  it  is  also  required  that  the  local  quadratic  approximation  to  the 
objective  function  be  minimized.  The  search  direction  determined  from 
(28)  or  (36)  does,  in  fact,  determine  a unique  point  which  is  the  exact 
solution  when  the  constraints  are  linear  and  the  objective  function  is 
quadratic.  In  fact,  the  accelerated  multiplier  method  of  Ref.  1 uses  a 
single  quadratic -linear  step  of  the  form  (36).  A principle  difference 
between  the  new  algorithm  and  that  of  Ref.  1 is  the  repeated  use  of  the 
quadratic -linear  steps  until  a point  satisfying  the  constraints  is  located. 

The  unconstrained  minimization  algorithm  requires  a specified 
basis  estimate  B and  specified  multiplier  estimates  X.  Estimates  of 
the  multipliers  are  obtained  by  minimizing  the  error  in  the  Kuhn-Tucker 
conditions.  The  multiplier  estimates  are  then  used  to  construct  an  esti- 
mate of  the  basis.  The  basis  determination  process  computes  multipliers 
and  constructs  a basis  estimate  at  any  point  x.  Details  of  the  process 
are  described  in  Ref.  I. 
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SECTION  4 


THE  GRADIENT  PROJECTION  MULTIPLIER  ALGORITHM 

The  basic  steps  of  the  gradient  projection  multiplier  algorithm  are; 

k . 

Step  1.  Lagrangian  Phase:  For  a fixed  basis  B , fixed  multipliers 
and  fixed  penalty  weight  minimize  the  augmented  objective 
function  (II)  using  the  unconstrained  optimization  algorithm  given  in 
Section  2.  Call  the  solution  x . 

Step  2.  Basis  Determination:  Keeping  x fixed,  compute  a new  basis 
B and  multipliers  X using  the  procedure  described  in  Ref.  1. 

Step  3.  Constraint  Phase:  Beginning  at  x with  the  fixed  basis  B, 
minimize  the  augmented  objective  function  (11)  with  L = 0 and  r = 1. 

Call  the  solution  If  P(x^^^)  0,  constraints  may  be  inconsistent. 

k+  1 

Step  4.  Basis  Determination:  Keeping  x fixed,  determine  a new 

basis  B^'*’^  and  multipliers  using  the  procedure  described  in 

k+1 

Ref.  1.  When  checking  for  inconsistent  constraints,  B must  be 
different  than  B;  if  not,  terminate. 

Step  5.  Convergence  Test:  e(x,  X)  <6^  and  O’ j.  <62  where  e(x,  X)  is 
the  absolute  error  in  the  Kuhn-Tucker  conditions  (6)  and  (8),  and  is  an 
estimate  of  the  resolution  error  in  the  variables  x and  X,  (cf.  Ref.  1), 

Step  6.  Penalty  Weight  Definition:  If  B^^  ^ = B = B^,  keep  penalty  weight 
unchanged,  i.  e.,  set  ^ = r^.  Otherwise,  increase  penalty  weight 
(cf.  Ref.  1). 

k_k+l  ,k_,k+l 

Step  7.  Update  Information:  Set  k - k+ 1,  x - x , K - K , 

B^  = B^^^,  etc.  Return  to  Step  1. 

A number  of  points  regarding  the  implementation  of  the 
overall  algorithm  now  deserve  clarification.  First,  observe  that 
current  gradient  and  Hessian  matrix  information  is  used  to 
initiate  the  different  operations.  For  example,  the  Hessian  matrices 
generated  by  the  unconstrained  algorithm  during  the  Lagrangian  phase 
can  be  used  to  initiate  the  penalty  minimization  in  the  constraint  phase. 
Secondly,  when  there  is  an  indication  of  inconsistent  constraints,  the 


fRECXiaKl  FiOS  blamc 


basis  determination  procedure  constructs  a new  basis  by  deleting  all 
satisfied  constraints  from  the  old  basis.  If  no  constraints  can  be 
! deleted,  the  algorithm  terminates  in  an  error  mode. 

[ The  philosophy  of  the  algorithm  is  to  alternately  satisfy  the 

I Lagrangian  condition  V L(x,  ^ ) = 0 and  the  constraint  condition, 

f During  the  constraint  phase  of  the  algorithm  the  search  directions  are 

computed  using  the  assumption  that  the  objective  function  is  quadratic 
and  the  constraints  are  linear,  resulting  in  the  projected  gradient 
directions.  The  penalty  weight  is  not  increased  once  the  correct  basis 
is  determined;  however,  it  is  allowed  to  increase  during  the  basis 
determination  process.  This  procedure  prevents  cycling  between  two 
or  more  incorrect  estimates  of  the  basic  set  of  constraints.  Further- 

I )Se 

more,  the  correct  basis  B is  usually  identified  quickly  because  of  the 
forced  increase  in  the  penalty  weight. 

I 

i 

I 

I 


! 


I 

i 


-18- 


SECTION  5 

NUMERICAL  EXPERIENCE 

The  algorithm  described  in  the  previous  section  has  been 
implemented  in  a digital  computer  program.  A rather  extensive  set  of 
test  problems  have  been  solved  using  a CDC  7600  digital  computer. 

This  section  presents  the  results  of  this  numerical  experience.  Most 
of  the  problems  have  been  drawn  from  the  literature,  and  it  is  felt 
that  the  mathematical  complexity  and  nonlinearity  is  fairly  representative 
of  the  kind  of  problems  encountered  in  practice.  However,  the  problems 
do  have  two  significant  attributes  which  are  distinct  from  those  commonly 
encountered  in  engineering  problems. 

First,  the  function  evaluation  process  is  relatively  inexpensive, 
and  errors  in  the  evaluation  process  are  on  the  order  of  the  machine 
accuracy.  Similarly,  accurate  gradients  can  be  obtained  cheaply  for 
the  test  problems.  For  practical  problems,  however,  function  evaluations 
can  be  quite  costly  and  may  contain  inaccuracies  significantly  larger 
than  the  machine  accuracy.  (A  typical  trajectory  optimization  example 
is  described  in  Ref.  14.)  Because  the  function  evaluations  for  these 
practical  problems  are  so  costly  with  respect  to  the  computational  ex- 
pense of  the  optimization  algorithm,  we  have  used  the  number  of  function 
evaluations  as  a measure  of  algorithm  effectiveness. 

Secondly,  the  required  gradient  information  is  obtained  by 
evaluating  corresponding  analytically  derived  partial  derivative 
expressions.  Practical  problems,  on  the  other  hand,  may  require 
that  gradient  information  be  evaluated  using  numerical  methods.  For 
this  reason,  no  attempt  has  been  made  to  present  an  "equivalent"  number 
of  function  evaluations,  because  such  a quantity  is  highly  dependent  upon 
the  numerical  differentiation  procedure.  Such  quantities  as  perturbation 
sizes  and  error  tolerances  can  greatly  influence  the  accuracy  of 
numerical  derivatives  and  consequently  obscure  the  overall  behavior  of 
the  optimization  process.  If  gradient  information  must  be  obtained 
numerically,  one  czm  expect  a two-fold  degradation;  (i)  because  each 
gradient  evaluation  will  require  one  or  more  additional  function  evaluations, 
and  (2)  because  inaccurate  gradient  information  may  necessitate  more 
optimization  iterations. 
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The  test  problems  given  in  the  appendix  have  been  organized 
into  four  separate  categories:  (1)  unconstrained,  (2)  nonlinear  least 
squares,  (3)  equality  constrained,  and  (4)  inequality  constrained.  The 
results  are  presented  in  Tables  1 through  4 in  condensed  form  for  each 
of  the  categories.  Specifically,  we  present  the  number  of  function  and 
gradient  evaluations  required  for  convergence.  For  example,  the  first 
problem  was  solved  by  evaluating  the  objective  function,  constraints,  and 
the  corresponding  gradient  vectors  at  four  points.  For  all  problems, 

_5 

convergence  is  defined  as  in  Ref.  1 with  6^  = 62  = 10  , which  guarantees 

five  significant  figures  accuracy  in  the  solution,  and  absolute  satisfaction 
of  the  constraint  and  Lagrangian  conditions  to  within  10  The  number 
of  cycles  of  the  algorithm  is  presented  for  the  constrained  examples.  It 
is  the  author's  opinion  that  despite  the  theoretical  elegance  of  a general 
mathematical  programming  algorithm,  one  can  not  espouse  its  numerical 
effectiveness  without  solving  a set  of  test  problems  at  least  as  broad  as 
those  given. 
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SECTION  6 


n 

I 

ri 


SUMMARY  AND  CONCLUSIONS 

I 

This  paper  describes  a gradient  projection-multiplier  method 
for  solving  the  general  nonlinear  programming  problem.  The  algorithm 
poses  a sequence  of  unconstrained  optimization  problems  which  are 
solved  using  a new  projection-like  formula  to  define  the  search  directions. 

Ill  conditioning  of  the  search  direction  computation  is  avoided  by  using 

an  expression  which  approaches  the  projected  gradient  direction  for  ^ 

large  penalty  weights.  The  unconstrained  algorithm  is  used  to  locate 

points  where  the  Lagrangian  condition  VL(x,  = 0 is  satisfied  by  ' 

minimizing  the  augmented  objective  function.  Points  satisfying  the 

constraints  are  located  by  applying  the  unconstrained  algorithm  to  the  | ' 

penalty  function.  New  estimates  of  the  Lagrange  multipliers  and  basis 

constraints  are  made  at  points  satisfying  the  Lagrangian  condition  and  j 

the  constraint  condition.  The  penalty  weight  is  increased  only  when  j ; 

necessary  to  prevent  cycling.  Although  we  do  not  prove  quadratic 

convergence,  numerical  experience  tends  to  confirm  this  assertion.  j 


• W 
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Table  1.  Unconstrained  Problems 


Problem  No. 

Function  and  Gradient  Evaluations 

1 

4 

2 

49 

3 

57 

4 

34 

5 

12 

6 

8 

, 7 

28 

8 

17 

1 ^ 

113 

' 10 

6 

11 

54 

Table  2. 

Nonlinear  Least  Squares  Problems 

1 Problem  No. 

Function  and  Gradient  Evaluations 

1 1 

6 

' 1 

27 

3 

1 1 

4 

9 

! 5 

2 

6 

15 

7 

6 

8 

22 

9 

12 

10 

10 

11 

3 

12 

7 

1 

13 

15 

i 


t ; 
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Table  i. 


Equality  Constrained  Problems 


Problem  No.  Function  and  Gradient 

Evaluations 

No.  of  Cycles 

1 

8 

1 

1 

2 

10 

1 

3 

21 

2 

4 

17 

2 

5 

17 

1 1 

1 

6 

19 

2 

1 

! 

7 

18 

2 1 

8 

16 

2 

9 

6 

1 

10 

6 a 

1 

11 

18b 

1 ' 

\ 

12 

27 

1 

1 

13 

188 

2 

14 

46 

2 

15 

60 

2 

16 

141 

2 

j 

17 

39 

1 

1 

a 

* 

Algorithm  terminated  at  a saddlepoint  (17P(x  ) = 0, 
P{x*)  /O,  and  P(x)  = 0 does  exist). 

although  ' 

i 

1 

1 

1 

b 

. 

Inconsistent  constraints  (P(x  ) = 0 does  not  exist). 

• 

1 

t 

) 

1 

1 

>1 

( 

( 

i 

! 

t 

-23- 

I 

1 

! 

i 


Table  4. 


Inequality  Constrained  Problems 


Problem  No. 


Function  and  Gradient 
Evaluations 


No.  of  Cycles 


I 


I 

I 

I 

I ' 

I 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 
1 1 
12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34  

a Constraint  gradients 


21 

5 

22  . 

28 

18 

58 

50 

50 

32 
28 
43 
43 
77 
27  a 
25 
17a 
40 
48 
21 
65 

13 

14 

33 
48 
38 

29 

81 

318 

9 

79 

86 

29 

43a 

2 ^ 

linearly  dependent  at  x . 
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1 

1 

1 

2 

1 

2 

4 

2 

3 

2 

1 

3 

3 

1 

2 

1 

3 

3 
2 

4 
1 
2 
2 
3 
2 
2 
1 

1 

1 

1 

1 

1 

1 


1 


I 


1 

< i 
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APPENDIX 


NONLINEAR  PROGRAMMING  TEST  PROBLEMS 

This  appendix  presents  the  set  of  test  problems  used  to  assess 
the  effectiveness  of  the  nonlinear  programming  algorithm  described. 

The  problems  are  organized  into  four  categories:  (I)  unconstrained 
problems,  (2)  nonlinear  least  squares  problems,  (3)  equality  constrained 
problems,  and  (4)  inequality  constrained  problems. 

When  no  information  is  given  to  the  contrary,  one  can  assume 
all  quantities  used  by  the  numerical  processes  are  scaled  (in  the 
sense  described  in  the  Appendix  of  Ref.  9)  in  the  range  -20  ^ x ^ 20. 

Unless  noted  otherwise,  the  initial  penalty  weight  is  r°  = 1,  and  the 
initial  basis  is  assumed  to  be  empty,  i.  e.  , B = ].  The  points  x 

presented  are  converged  values  where  convergence  is  defined  in  Section  7 of 

_5 

Ref.  1,  with  6^  = 62  = 10  . When  the  exact  solution  is  known,  its  value 

is  presented  following  the  computationally  obtained  value.  To  conserve 
space,  problem  statements  appearing  elsewhere  in  consistent  notation 
are  merely  referenced. 
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UNCONSTRAINED  PROBLEMS 


A.  1 

1,  Ref. 


i 

2.  Ref. 


3.  Ref. 


15 


2 2 
Minimize  ~ ^ ^^2 

-50<x<50 
x°  = (4.2) 

X*  = (2.  1316  X 10"^'^,  1.0658  X 10"^^)  ; (0,0) 
f = 2.2719  X 10  ® ; 0 


no  scaling. 


2 

Minimize  f(x)  = 100  (x2  - Xj  ) + (1  - Xj) 

x°  = (-1.2,1) 

X*  = (1.,  1.)  ; (1, 1) 

f*  = 6.6421  x 10"^®  ; 0 

no  scaling. 


15 

Minimize  “ (^1  ■*"  ^ 5(^3  “ *4)^ 

+ (X2  - 2x3)^^  + 10(xj  - x^)^ 

-50£x£50 

x°  = (3,-1,  0,1) 

X*  = (1. 8159  X lO"'^,  -1.8159x10"^,  9.398  x 10'^,  9.  3980  x 10"^) 

; (0,0, 0,0) 

f*  = 2.3943  x 10"^^  ; 0 

no  scaling. 
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4.  Ref.  1 5 


Minimize  ~ ^ (f'U  J ^ ^3 


1 

I 


where 


Z IT  6 = { 


-1 


tan  (x^/x^)  X|>0 


TT  + tan  ^(x^/xp  Xj<  0 


, 2 ^ 2,7 

r = (xj  ^2  ^ 


-5  < x^,  X2S  5 
-2.  5<x^<7.  5 
x°  = (-1,0,0) 

X*  = (1.,  5.2171  X 10"^°,  6.0920  X 10‘^°)  ; (1,0,0) 

f*  = 7.9033  x 10"^®  ; 0 


no  scaling. 


5.  Ref.  17 


Minimize  f(x)  = (x^  “ x^  ) + (1  - x^ 


-100<x<100 


x'"  = (-2,-2) 

X*  = (9.  9999  x 10"^  9.9999  x 10'*)  ; (1,1) 

f = 2.4809  X 10  ° ; 0 

no  scaling. 
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-100<x<100 
x°  = (1.5) 

:c*  = (1.,  1.)  ; (1.  1) 

f*  = 1.0172  X 10"^^  ; 0 

no  scaling. 

7.  Ref.  17 

32  2 

Minimize  f{x)  = 100  (X2  " ) "*  (I'X^) 

-100<x<100 
x°  = (-1.2,1) 

x*  = (9.  9999  X 10"\  9.  9999  X lO'S  ; (1,1) 

f*  = 2.0912x10"^^  ; 0 

no  scaling. 

8.  Ref.  18 

3 r i T 2 

Minimize  f(x)  = ^ j^a^  - x^ 

where 

a^  = 1.  5 , a^  = 2. 25  , ^3  “ 2.625 

-100<x<100 
x°  = (8..  2) 

x"^  = (3.  .5)  ; (3,.  5) 

* -20 
f = 1.7257  X 10  ; 0 

no  scaling. 
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1 1,  Ref.  21 


2 


Minimize  f(x)  = / ^ L exp  - - .Oli  J 

i=l 


■1 


2/3 


where  = 25  + (-50  In  . Oli) 


. 1 « x^  s 101 
0 ^ x^  i 25.  6 
0 ^ Xj  ^ 5. 
x°  = (100,  12.  5,  3) 

X*  = (50.  25,  1.5) 
f*  = .52009  x 10"^"^;  0 


I 


i 

I 


] 

I 

I 


I ! 
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A.  2 NONLINEAR  LEAST  SQUARES  PROBLEMS 

Problems.  1-10;  Problems  1-10  in  Ref  10.  | 

Problem  11.  Ref.  19 

21  2 

Minimize  f(x)  = E (x) 
i=l 

where  = i - 1 

2 3 4 5 I 

Yi  = 1 + ti  + ti  + ti  + ti  + ti 

23^45^ 

Ci(x)  = yi  - (xi  + X2t.  + X3t.  + x^t.  + X5tj  + x^t.  ) 

I 

O 'i 

x^=0  i=l,...6  j'l 

X*  = (1.00000,  1.00000,  1.00000,  1.00000,  1.00000,  1.00000) 

f*  = . 13777  X lO"^'* 

No  scaling 

Problem  12. 

2 

Minimize  f(x)  = E (x) 

i=l 

where  c^(x)  = { y.  - [x^u.  + exp  (x^v-)]  3 

and  u^,  v^,  w^,  and  are  given  in  Table  A-1.  i 

, 
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15.  Ref.  21 


where 


10 

Minimize  f(x)  x. 

i=l 


= -6.089 

a2  = 17. 164 

a^  = -34.  054 

= -5.914 

ag  = -24. 721 

ag  = -14.  986 

= -24. 100 

ag  = -10.708 

a^  = -26.  662 

= -22. 179 

Cj(x)  = x^  + Zx^  + 2x^  + + Xjg  -2=0 

C2(x)  = x^  + 2xg 

Cjtx)  = X3  + x.^  + Xg  + 2Xg  + XjQ  -1  = 0 


1.  X 10"^  s X s 10 
xP  = .l  i=l,...,10 

X*  = (.40668  X 10"\  .14773,  .78315,  .14142  x 10"^,  .48524, 

. 69317  X 10'^,  . 27399  x 10■^  . 17  947  x 10■^  . 37314  x 10 
. 96871  X lO'S 
f*  = -47.761 
no  scaling. 


iJ 

! I 

•i 

n 


i 


y 
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16.  Ref.  21 


10  10 

Minimize  f(x)  |e^i  [ e’^j)  ]} 

i=l  ' j = l 

where  the  are  given  in  Problem  15. 

Cj^(x)  = e^l  + 2e^2  + 2e^3  + e^6  + -2  = 0 

C2(x)  = e^4  + 2e^5  + e^6  + q^7  -1=0 
c^lx)  = e^3  + e^7  + e^8  + 2e^9  + e^l0-l  = 0 
-100  ^ X s 100 
x°  = -2.  3 i = 1 .10 

i 

x*  = (-3.2024,  -1.9123,  -.24441,  -15.670,  -.72166, 
-7.2736,  -3.5965,  -4.0206,  -3.2885,  -2.3344) 

f*  = -47.760 

Constraint  scaling:  = 10,  i = 1,  2,  3. 

17.  Ref.  21 

2 2 2 

Minimize  f(x)  = 1000  - x^  - 2x  2 - x^  - Xj^X2  - 
2 2 2 

c , (x)  = x,  +x_  + x -25  = 0 
1 123 

~ 8x^  + 14x2  ^ ^^^3  “ ~ 0 

0 « X s;  100 

x°  = (2,  2,  2) 

X*  = (3.  5121,  . 21698,  3.  5522) 
f*  = 961.715 
No  scaling. 
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A.  4 INEQUALITY  CONSTRAINED  PROBLEMS 

Problems  1-26:  Problems  1-26  in  Ref  1. 

27.  Problem  27  in  Ref.  1 except  with 

B°=  [9] 

Constraint  scaling: 

Wo  = 10-^ 


w.  = 1 i = 1,  2.  3,  4,  5,7 
1 


w^  = 10 


Wo  = w-  = 100 


w.  = 10"^  i = 10,  11,  12,  13,  14,  15 

i 


28.  ilef.  20 


15  5 5 5 2 

Maximize  f(x)  = T c x-  - E E y..  x.  x.  - 2E  6^  x. 

i=6  j=l  i=l  ^ ^ i=l 


c^(x)  = x^  ^ 0 

~ ^j-15  ^ ^ j-15 


i = 1, ...  15 


+ 2E  y.  . , c X. 


i=l 


i,  j-15  -i 


15 

-E  a. 


i=6 


i-5,  j-15  -i 


X.  ^ 0 j = 1 6,  . . . 20 . 


where  the  coefficients  O,  y,  6,  and  e are  given  in  Problem  17  of  Ref  1. 


•10  « X.  « 20 
1 


• 100  s xi2  ^ 100 


i = 1 11 


.1 
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x‘‘  = (.  30000,  .33346,  .40000,  .42832,  .22397, 


0..  -.56843  x 10"^^,  5.1740,  0.,  3.06111. 
11. 8396,  0. , 0. , . 10391,  0. ) 
f*  = -32.  3486 
Constraint  scaling: 

= 10  for  i = 6,  12,  13 

= 1 otherwise 

29.  Ref.  21 

2 2 

Minimize  f(x)  = (Xj^  - 2)  + (x^  - 1) 

Cj(x)  = Xj  - 2x2  ^ ® 

^ 2 

C2(x)  = -^-X2  +1  ^0 

4 

-10  s X s 10 
x°  = (2,  2) 

X*  = (.  82287,  . 91143) 
f*  = 1.  3934 
no  scaling. 

i 

I • 

j 
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+ . 030234  XjX^  -1.  28134  X 10‘^  x^Xj 

-5^-7^ 

+ 3.  5256  X 10  ^2^1  "^•266  x 10  ^2^1  i 

•>  -3  ^ -5  ^ 

+ . 2564  5 x^  -3.  4604  xlO  x^  +1.3514x10  x^ 

_1  -6  2 ^ 

-(28.  106)  (x^  + 1)  -5.  2375  x 10  ° x^  ) 

-ft  3 ? - 1 f)  3 ^ 

-6.  3 X 10  x^  x^  + 7 x 1 0 Xj  x^  ^ 

-4  2 -63 

+ 3. 4054  X 10  ^1^2  ” ^^38  x 10  ^^1^2  ' 

-2.  8673  e-xp  (.  0005  x^x^) 


Cj(x)  = 

X1X2 

- 700  S: 

0 

C2(x)  = 

^2  " 

’{+) 

1 - 

C-^{x)  = 

(X2  - 

50)2  _ 

5(x^  - 55)  s 

11 

u 

75  - 

Xj  2 0 

c^(x)  = 

65  - 

X2  ^ 0 

0 

S X ^ 

200 

x°  = (90,  10) 

X*  = (7  5,  65) 
f*  = 58.903 

I 

no  scaling.  J 
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31. 


Ref,  ZO.  p.  31.  (Also  Ref.  Zl,  p.  401) 


i 

a 

1 

A 

1 

a . 

X 

A 

Vi 

1 

. 1 

. 00189 

1 1 

10 

. 702 

2 

1 

. 1038 

12 

11 

. 528 

3 

2 

. 268 

13 

12 

. 385 

4 

3 

. 506 

14 

13 

. 257 

5 

4 

. 577 

15 

14 

. 159 

6 

5 

. 604 

16 

15 

. 0869 

7 

6 

. 725 

17 

16 

. 0453 

8 

7 

. 898 

18 

17 

.01509 

9 

8 

. 947 

19 

18 

. 00189 

10 

9 

. 845 

33.  Ref.  21 

Maximize  f(x)  = . 5 ^ ^3’^9  ” ^^5^9  ^ ^5^8  ” ^6^7^ 


Cj(x) 

= 1 

■ ^^3 

- X ^ 

0 

C2(x) 

= 1 

2 

- Xg 

2:  0 

Cj(x) 

= 1 

2 

-’^5 

2 

■ ’^6  ^ 

0 

= 1 

2 

■ ’^l 

- (X2  - 

Xg) 

^ 0 

c^(x) 

= 1 

- (’^l 

- X5) 

- (^2 

c^{x) 

= 1 

- (’‘l 

- x^) 

- (X2 

Cj(x) 

= 1 

- (X3 

- X3) 

- (X4 

- -6)"  ^ ° 
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Cg(x)  = 1 - (Xj  - x^)^  - (x^  - Xg)^  i 0 
Cg(x)  = 1 - x^  ~ ^^8  ' ^ ° 

cio(x)  = xjx^  - X2X3  ^ 0 


Cj^^(x)  = XgXg  ^ 0 
Cj2W  = - XgXg  ^ 0 

Ci3(x)  = XgXg  -xgx^  ^ 0 

-2  s X ^ 2 


